Hybrid Algorithms for Efficient Cholesky Decomposition and Matrix Inverse using Multicore CPUs with GPU Accelerators

The use of linear algebra routines is fundamental to many areas of computational science, yet their implementation in software still forms the main computational bottleneck in many widely used algorithms. In machine learning and computational statistics, for example, the use of Gaussian distributions is ubiquitous, and routines for calculating the Cholesky decomposition, matrix inverse and matrix determinant must often be called many thousands of times for common algorithms, such as Markov chain Monte Carlo. These linear algebra routines consume most of the total computational time of a wide range of statistical methods, and any improvements in this area will therefore greatly increase the overall efficiency of algorithms used in many scientific application areas. The importance of linear algebra algorithms is clear from the substantial effort that has been invested over the last 25 years in producing low-level software libraries such as LAPACK, which generally optimise these linear algebra routines by breaking up a large problem into smaller problems that may be computed independently. The performance of such libraries is however strongly dependent on the specific hardware available. LAPACK was originally developed for single core processors with a memory hierarchy, whereas modern day computers often consist of mixed architectures, with large numbers of parallel cores and graphics processing units (GPU) being used alongside traditional CPUs. The challenge lies in making optimal use of these different types of computing units, which generally have very different processor speeds and types of memory. In this thesis we develop novel low-level algorithms that may be generally employed in blocked linear algebra routines, which automatically optimise themselves to take full advantage of the variety of heterogenous architectures that may be available. We present a comparison of our methods with MAGMA, the state of the art open source implementation of LAPACK designed specifically for hybrid architectures, and demonstrate up to 400% increase in speed that may be obtained using our novel algorithms, specifically when running commonly used Cholesky matrix decomposition, matrix inverse and matrix determinant routines. Original Contributions The main contributions in this thesis are a collection of optimised algorithms that may be applied to general blocked linear algebra routines on hybrid and heterogenous architectures, such as systems with a multicore CPU and GPU accelerator, and systems consisting of multiple GPU accelerators. The first contribution is a new automated approach for blocked linear algebra routines that allows a new level of dynamic blocking to balance the workload more efficiently between heterogenous CPU and GPU computing devices, which have varying clock speeds and very different memory capacities. The second contribution considers the problem of transferring diagonal submatrices between CPU memory and GPU memory, which is an essential operation in many blocked linear algebra routines. We develop a novel algorithm for achieving this transfer efficiently and demonstrate the resulting improvement in speed. The third contribution is an original method for running multiple GPU kernel functions simultaneously on GPUs that do not have inherent hardware support for this capability. In these cases, a large number of GPU processors may often be left idle, waiting for a single kernel to complete. We demonstrate our algorithm using an example whereby a Cholesky decomposition kernel may be run concurrently with a matrix multiply kernel, achieving much higher performance and efficiency than previously possible on the same hardware using existing state of the art linear algebra libraries. We employ the Cholesky decomposition, matrix inverse and determinant operations as motivating examples, and demonstrate up to a 400% increase in speed that may be obtained using combinations of the novel approaches presented.